Skip to content

Profiler CLI #1623

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 246 commits into from
Nov 7, 2025
Merged

Profiler CLI #1623

merged 246 commits into from
Nov 7, 2025

Conversation

@sundarshankar89
Copy link
Collaborator

@sundarshankar89 sundarshankar89 commented Jun 3, 2025 •
edited
Loading

Changes

What does this PR do?

This introduces the intial version of CLI

Relevant implementation details

Caveats/things to watch out for when reviewing:

Linked issues

~ ❯ databricks labs lakebridge execute-database-profiler --help                                                                                                                                                 26s
Profile the source system database

Usage:
  databricks labs lakebridge execute-database-profiler [flags]

Flags:
  -h, --help                 help for execute-database-profiler
      --source-tech string   (Optional) The technology/platform of the sources to Profile

Global Flags:
      --debug            enable debug logging
  -o, --output type      output type: text or json (default text)
  -p, --profile string   ~/.databrickscfg profile
  -t, --target string    bundle target to use (if applicable)

~ ❯ databricks labs lakebridge execute-database-profiler --source-tech mssql
12:24:59    ERROR [d.labs.lakebridge] Only following source system is supported ['synapse']
12:24:59    ERROR [d.l.lakebridge.execute-database-profiler] ValueError: Invalid source technology mssql


~ ❯ databricks labs lakebridge execute-database-profiler --source-tech Synapse
[UPGRADE ADVISED] Newer lakebridge version was released 12 days ago. Please run `databricks labs upgrade lakebridge` to upgrade: feature/profiler_entry_point -> v0.10.11
12:25:15    ERROR [d.l.lakebridge.execute-database-profiler] ValueError: Connection details not found. Please run `databricks labs lakebridge configure-database-profiler` to set up connection details for synapse.
~ ❯

~ ❯ databricks labs lakebridge execute-database-profiler
Select the source technology
[0] synapse
Enter a number between 0 and 0: 0
12:32:50    ERROR [d.l.lakebridge.execute-database-profiler] ValueError: Connection details not found. Please run `databricks labs lakebridge configure-database-profiler` to set up connection details for synapse.

Functionality

  • added relevant user documentation
  • added new CLI command
  • modified existing command: databricks labs remorph ...
  • ... +add your own

Tests

  • manually tested
  • added unit tests
  • added integration tests

sundarshankar89 and others added 30 commits March 21, 2025 19:39
@sundarshankar89
pipeline extension
e61dd3e
@sundarshankar89
Init commit
bfa8490
@sundarshankar89
Merge branch 'main' into feature/synapse_data_collector
65bd96f
@sundarshankar89
Merge branch 'main' into feature/synapse_data_collector
e578178
@sundarshankar89
Merge branch 'main' into feature/pipeline_extensions
73b36c2
@sundarshankar89
Init commit
8e95927
@sundarshankar89
Merge remote-tracking branch 'origin/feature/synapse_data_collector' …
cabf93a 
…into feature/synapse_data_collector
@sundarshankar89
Merge branch 'feature/pipeline_extensions' into feature/synapse_data_…
ec33f67 
…collector
@sundarshankar89
Merge branch 'main' into feature/pipeline_extensions
440cc70
@sundarshankar89
Merge remote-tracking branch 'origin/feature/synapse_data_collector' …
d6fcabb 
…into feature/synapse_data_collector
@sundarshankar89
fmt fixes
2988569
@sundarshankar89
fmt fixes
f0c87d6
@sundarshankar89
refactor
0764e01
@sundarshankar89
Added tests for dependency
d188705
@sundarshankar89
Added tests for dependency
68b0f83
@sundarshankar89
Added Failure tests
5bd9408
@sundarshankar89
installer as venv
5899a3c
@sundarshankar89
fmt fixes
3cdbf7e
@sundarshankar89
Merge branch 'feature/pipeline_extensions' into feature/synapse_data_…
822da6b 
…collector
@sundarshankar89
fmt fixes
299d293
@sundarshankar89
Added Python Script
9186b89
@sundarshankar89
Added Python Script
18f9b42
@sundarshankar89
Fmt fixes
877a239
@sundarshankar89
Merge branch 'main' into feature/pipeline_extensions
492a12f
@sundarshankar89
Merge branch 'feature/pipeline_extensions' into feature/synapse_data_…
d0ca03f 
…collector
@sundarshankar89
intermediate commit
603aa0f
@sundarshankar89
init commit
204c639
@sundarshankar89
Merge branch 'main' into feature/synapse-cred-configuration
1af4de9
@sundarshankar89
Merge branch 'main' into feature/pipeline_extensions
75fbfb5
@sundarshankar89
fmt fixes
de87a2a
gueniai
gueniai approved these changes Oct 28, 2025
View reviewed changes
Copy link
Collaborator

@gueniai gueniai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

goodwillpunning
goodwillpunning approved these changes Nov 6, 2025
View reviewed changes
Copy link
Contributor

@goodwillpunning goodwillpunning left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@goodwillpunning goodwillpunning mentioned this pull request Nov 7, 2025
Merged
7 tasks
@gueniai gueniai enabled auto-merge November 7, 2025 17:07
@gueniai gueniai added this pull request to the merge queue Nov 7, 2025
github-merge-queue bot pushed a commit that referenced this pull request Nov 7, 2025
@goodwillpunning @gueniai
Add Create Profiler Dashboard Command to the CLI (#2134)
09bd34e 
### What does this PR do?

This PR adds a `create-profiler-dashboard` CLI command. Once executed,
the command will deploy a profiler summary as a Databricks dashboard to
the user's workspace.

### Relevant implementation details

The `create-profiler-dashboard` CLI command performs 3 actions:
1. Uploads the profiler extract to a UC volume
2. Deploys a workflow for ingesting the extract file and converting to
Delta tables
3. Deploys a summary dashboard based on the source technology that was
profiled

### Caveats/things to watch out for when reviewing:

The job can be uninstalled when a Lakebridge `uninstall` command is
executed.

### Linked issues

Compliments #1623

### Functionality

- [ ] added relevant user documentation
- [X] added new CLI command
- [ ] modified existing command: `databricks labs lakebridge ...`
- [ ] ... +add your own

### Tests

- [X] manually tested
- [ ] added unit tests
- [ ] added integration tests

---------

Co-authored-by: Guenia Izquierdo <[email protected]>
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to a conflict with the base branch Nov 7, 2025
@gueniai gueniai enabled auto-merge November 7, 2025 19:10
@gueniai gueniai added this pull request to the merge queue Nov 7, 2025
Merged via the queue into main with commit 27a41a5 Nov 7, 2025
9 of 10 checks passed
@gueniai gueniai deleted the feature/profiler_entry_point branch November 7, 2025 20:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

Copilot code review Copilot Copilot left review comments

@gueniai gueniai gueniai approved these changes

@goodwillpunning goodwillpunning goodwillpunning approved these changes

@m-abulazm m-abulazm Awaiting requested review from m-abulazm

@asnare asnare Awaiting requested review from asnare

Assignees

@sundarshankar89 sundarshankar89

Labels

feat/cli actions that are visible to the user feat/profiler Issues related to profilers

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@sundarshankar89 @asnare @gueniai @goodwillpunning @m-abulazm